Skip to content

feat(dynamo)!: serve NeMo RL's own vLLM rollouts through Dynamo's backend SDK - #4905

Merged
terrykong merged 42 commits into
NVIDIA-NeMo:mainfrom
jthomson04:jthomson04/dynamo-native-engine
Nov 11, 2026
Merged

terrykong merged 42 commits into
NVIDIA-NeMo:mainfrom
jthomson04:jthomson04/dynamo-native-engine

Conversation

@jthomson04

@jthomson04 jthomson04 commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

policy.generation.dynamo: true on a SingleController vLLM generation config is now the whole Dynamo switch. NeMo RL's own vLLM workers register their AsyncLLM with a managed dynamo.frontend. They do this through Dynamo's engine-agnostic backend SDK (dynamo.common.backend), from inside each worker's actor loop.

The engine is the native vLLM worker, not Dynamo's dynamo.vllm, so:

  • Features come with the native worker. Refit (every transport), pause, quantization, speculative decoding, parallelism and metrics are the native backend's own. They work under Dynamo without porting.
  • Shared code, not copies. The engine serves each rollout with the worker's own _serve_rollout (refactor(vllm): share the per-sample rollout pipeline and chat token helpers #4978), the same method native generate_async calls, and the driver assembles the output with vllm/request_pipeline.py.
  • One Dynamo side for every backend. dynamo/engine_adapter.py (DynamoEngineAdapter) owns the request kinds, the error mapping and the completions-only registration, and imports no backend. dynamo/vllm_adapter.py (DynamoVllmAdapter) is the vLLM backend: serve_rollout, start, abort and kv_event_sources. feat(dynamo): serve NeMo-Gym through the Dynamo frontend #4979 adds a request kind and Megatron (feat(megatron): serve NeMo-Gym from the engines behind the Dynamo frontend #5097) adds a backend without touching the Dynamo side.
  • Dynamo code cannot block a vLLM bump. NeMo RL imports only Dynamo's backend SDK, which imports no vLLM; unit guards enforce both. The one remaining wire coupling is vLLM's KV-event format, and the Dynamo functional overlays fail if the router applies no events. Dynamo's dependency bounds still can (ai-dynamo-runtime caps pydantic at 2.13): until a Dynamo release lifts a bound, override it under [tool.uv] override-dependencies.

This replaces #4725 and #4822. P/D disaggregation (#4773) is out of scope.

Split at review request (three PRs, stacked):

  1. refactor(vllm): share the per-sample rollout pipeline and chat token helpers #4978 (merged): the vLLM request-pipeline refactor this PR builds on.
  2. This PR: Dynamo serving for NeMo RL's own rollouts, on the frontend's /v1/completions. It does not need a Gym bump.
  3. feat(dynamo): serve NeMo-Gym through the Dynamo frontend #4979: NeMo-Gym through the Dynamo frontend, together with the engine's general OpenAI path (chat, parsers and templates, media, guided decoding, prompt logprobs), since Gym is its only client. It waits on feat(vllm_model): Dynamo frontend dialect and dynamo_worker capture handler Gym#4817.

What changes for users

  • Turning it on: backend: vllm + dynamo: true. dynamo_cfg is optional; every field has a default (see examples/configs/grpo_math_1B_dynamo_single_controller.yaml).

  • Removed:

    • backend: dynamo, DynamoGeneration, the dynamo.vllm launcher, and the HTTP refit routes, token wrapper and metrics scraper;
    • the isolated /opt/dynamo_venv;
    • the dynamo_cfg keys engine, control_timeout_s, metrics_{include,exclude}_prefixes, worker_args and frontend_args.tokenizer*.

    Each is rejected at startup with the migration.

  • Packaging:

    • ai-dynamo and ai-dynamo-runtime 1.5.1 join the vllm extra (base package only, never ai-dynamo[vllm]).
    • Dynamo's planner and simulator dependencies (aisimulate, which brings in jax/flax, and kubernetes) are dropped with overrides.
    • ai-dynamo-runtime caps pydantic at 2.13, so vLLM venvs resolve pydantic 2.13.0 while the rest of the lock stays on 2.13.5.
    • BUILD_DYNAMO=1 now installs only etcd and nats-server.
  • NeMo-Gym with dynamo: true is rejected at startup until feat(dynamo): serve NeMo-Gym through the Dynamo frontend #4979.

  • dynamo_cfg.request_timeout_s defaults to no deadline, as native has none. On the SingleController, async_rl.rollout_failure.native.generation_timeout_s bounds both paths.

Rejected at startup with dynamo: true

  • colocated generation (a sleeping engine stays registered);
  • SingleController fleet health (a restarted shard does not re-register; follow-up);
  • NeMo-Gym (env.should_use_nemo_gym), until feat(dynamo): serve NeMo-Gym through the Dynamo frontend #4979;
  • expert_parallel_size > tensor_parallel_size (data parallelism inside vLLM, which Dynamo does not serve yet: above DP rank 0, vLLM refuses each engine's ipc:// KV-event endpoint);
  • vllm_cfg.http_server_api_key_required (the frontend serves the same engines on 0.0.0.0 and checks no bearer token);
  • frontend_args.router_mode: direct (it needs a worker ID that rollouts never carry);
  • dynamo: true on a non-vLLM backend;
  • dynamo_cfg without dynamo: true.

Validation

These local runs on 2x RTX PRO 6000 (sm_120) were made before the split, on the combined branch. Each is the native SingleController test unchanged, with and without ++policy.generation.dynamo=true; the Gym rows live in #4979.

Test Result, native vs Dynamo
grpo_dp_single_controller.sh (math, 2 steps) rewards identical; gen_kl_error 0.0006 vs 0.0005; token_mult_prob_error 1.014 both
grpo_dp_single_controller_reload_refit.sh rewards and generated tokens identical; token_mult_prob_error 1.0145/1.0131 vs 1.0151/1.0109

At 1e73994f85, the same math overlay locally:

  • served its rollouts through Dynamo as before: one engine registered for completions only, 8 rollouts routed, 200 KV events applied, and the services stopped cleanly;
  • then failed at the training step on No module named 'megatron' in the local driver venv. The native run fails identically there, so the local env can no longer finish the run; CI's L1 covers it.

Several engines and the KV router's benefit come from a separate stack. The Oct 28 bench on tyche used ai-dynamo 1.6.1, the #4955/#4979 stack, NeMo-Gym's OpenAI path, and 3 vLLM TP1 engines plus 1 trainer:

  • rollout throughput: native 0.077 vs Dynamo KV router 0.288 rollout groups/s;
  • prefix-cache hit rate: 49% vs 89%;
  • run-to-run noise: about 1%.

It shows several engines registering, publishing KV events and being routed between. It does not cover this PR's own rollout path or ai-dynamo 1.5.1: both CI runners have 2 GPUs, and two engines need 3.

At scale, the production Nano 3.5 RLVR recipe ran 10 steps on 86 nodes with dynamo: true, on the #4979 stack (the recipe uses NeMo-Gym) and ai-dynamo 1.5.1. It showed no meaningful step-time degradation compared with the recipe's native vLLM runs, and rewards tracked native (W&B o5ag5a9z for Dynamo, 6be0a52p and ids5h9lt for native).

Both paths fail fast when a generation worker is killed: a local chaos run with fleet health off stopped 7 s after the kill in both modes.

Dynamo results vary slightly run to run. Requests arrive over HTTP, so batch composition differs between runs; native repeats its first step bit for bit.

CI: L1_Functional_Tests_Dynamo.sh runs these as overlays:

  • SingleController math;
  • reload refit.

#4979 adds the three Gym overlays: token capture, no capture, and R3 with deferred routes.

Each overlay runs the native test's own metric checks plus check_dynamo_overlay.sh, which checks:

  • engines registered;
  • requests routed;
  • KV events applied;
  • services stopped.

Labels: this needs CI:L1. The Dynamo fast entries cannot pass on main's current image until it is rebuilt with /opt/dynamo/bin.

Follow-ups

  • Fleet health with Dynamo (restart re-registration, rollouts through the frontend).
  • Native deferred R3 drops routes on MoE models with dense layers. route_assembly.py accepts only MoE-layer payloads, while vLLM reports every layer. This happens identically on native and Dynamo.
  • Dynamo on the GB200 lane (the image needs BUILD_DYNAMO).
  • A vLLM bump drill, and throughput at scale.
  • Log at ERROR when an engine's Dynamo endpoint stops after registration (today only the registration wait checks it).
  • Dynamo upstream: an opt-out for configure_dynamo_logging() in embedded engines.

🤖 Generated with Claude Code

…end SDK

`policy.generation.dynamo: true` on a vLLM generation config is now the whole
Dynamo switch. NeMo RL's own vLLM workers register their AsyncLLM with a
managed dynamo.frontend through Dynamo's engine-agnostic backend SDK
(dynamo.common.backend), from inside each worker's actor loop. No Dynamo-owned
vLLM code runs, so refit, pause, quantization, speculative decoding and
parallelism are the native backend's own, and a vLLM bump never waits on
Dynamo.

- One request pipeline (vllm/request_pipeline.py) serves native
  generate_async, the native OpenAI server and the Dynamo adapter: the
  sampling-profile check, clamps, R3, token fields, capture, prefix splice
  and media.
- Rollouts go through the frontend's KV-aware router. The engine publishes
  KV events, and functional overlays fail if the router applies none.
- NeMo-Gym calls go through the frontend in Gym's Dynamo dialect
  (NVIDIA-NeMo/Gym#4817), with engine-side token and media capture.
- Startup rejects what Dynamo cannot serve: colocated generation, fleet
  health, the generation router, a bearer token, R3 with Gym but no capture,
  parser plugins and chat settings with no frontend counterpart.
- Removed: backend: dynamo, DynamoGeneration, the dynamo.vllm launcher, its
  refit routes, token wrapper, metrics scraper and isolated venv. Old keys are
  rejected with the migration.
- ai-dynamo and ai-dynamo-runtime 1.5.1 join the vllm extra, without the
  planner deps. BUILD_DYNAMO installs only etcd and nats-server.
- CI: the Dynamo lane runs native SingleController tests as overlays: math,
  Gym token capture, reload refit, and R3 with deferred-route capture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04 jthomson04 added the CI:L1 Run doctests, unit tests, and functional tests label Oct 24, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 24, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added Documentation Improvements or additions to documentation CI Relating to CI labels Oct 24, 2026
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 27791a4 (PR #4905 from jthomson04/dynamo-native-engine)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Run and CI breaks:
- Each engine publishes KV events on its own abstract ipc:// socket, so
  engines on one node can no longer draw the same port (C1).
- Engines get an OS-assigned DYN_SYSTEM_PORT, so a frontend on another node
  can read a custom chat template (C2).
- The Gym capture test skips without nemo_gym (C3).
- Gym calls apply the generation config's max_tokens only when the request
  leaves it unset, as vLLM's server does (C4).
- Lint: minimized swe1 recipe (C5) and import order (C6).

Behavior:
- vLLM client errors reach the client as InvalidArgument with their message,
  not as a bare 500 (C7).
- Non-capture Gym calls return routed experts with R3 on, and Gym#4817
  (now a369e28) lifts them, so R3 + Gym no longer needs token capture (C8).
- A disconnected capture call fails instead of being staged (C9).
- EP > TP is rejected at startup with Dynamo (C10).
- The KV-event block size matches vLLM's main-attention kinds (C17).

Messages and docs: timeout wording and error (C11, C12), missing etcd/NATS
advice (C13), dependency bounds (C19) and ray.sub's port layout (C20).

Tests: a Gym-without-capture Dynamo overlay in L1 (C14), the rollout retry
loop (C15), media dispatch and failure mapping (C16), hybrid block size
(C17) and the Gym engine_data key contract (C18).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The design holds up, and most of the 20 inline comments are small. Four should be fixed before this leaves draft, because each breaks a run or CI:

  1. (C1) Engines on one node pick their KV-event port at random and can collide; the second engine then fails to start (24% of starts at 8 engines per node). An abstract ipc:// socket per engine removes the port.
  2. (C2) A custom chat template stops a multi-node run from ever serving: the frontend cannot read a template file that exists only on the engine nodes. DYN_SYSTEM_PORT=0 lets each engine serve it.
  3. (C3) The capture test imports nemo_gym without a guard, so all three L0_Unit_Tests_Vllm_* shards now fail at collection.
  4. (C4) Gym calls are capped by the model's generation-config max_new_tokens even when they set max_tokens, which vLLM's own server does not do.

Two more fail CI's lint as soon as it runs: C5 (configs-minimize-check) and C6 (import order). The rest are lower-severity bugs (C7, C9), usability asks (C8, C10, C12, C13), docs (C11, C19, C20) and test gaps (C14–C18).

What works well

  • dynamo: true as a flag on the vLLM config, not a backend value, is the right shape. It cuts the "dynamo" literals outside the Dynamo package from 21 in 9 files to 5, and the conflicting settings are rejected at the one place every entrypoint calls (generation/__init__.py#L158-L169).
  • The request path shares code with native instead of copying it: off_policy_sampling_error, chat_generation_tokens and chat_routed_experts, splice_prefix_tokens with an EOS-count input, and _rollout_max_new_tokens / _rollout_sampling_params. Native generate_async is about 230 lines shorter for it.
  • The Dynamo L1 tests are the native SingleController tests plus ++policy.generation.dynamo=true and one shared check script, so Dynamo coverage moves with native coverage.
  • The decoupling guard: an import allow-list that runs in every lane (test_dynamo_vllm_decoupling.py#L63) and a fresh-interpreter "the SDK imports no vLLM" check in the vLLM lane (#L78). Both fail rather than skip.
  • Narrow seams: ManagedDynamoRuntime now owns only the services and takes a check_engines callback, so a failed registration fails with its cause; ROLLOUT_ANNOTATION sits driver-side, so the driver never imports ai-dynamo.
  • The engine tests drive a real vLLM worker over a fake AsyncLLM, through the native helpers it shares, with no GPU.
  • All three critical fixes from the #4822 review are here and covered: the top_k pin (reverting it fails a test), the f32 comparison of temperature and top_p, and the echoed-prefix splice.

PR description

  • The "One request pipeline" bullet overstates request_pipeline.py: it holds only the rollout result path (RolloutResult, selected_logprobs, rollout_result, rollout_output and the wire codecs), and the native OpenAI server does not import it. Suggested text: "Shared code, not copies. Rollouts share vllm/request_pipeline.py (result, R3 routes) and the worker's budget and sampling with native generate_async. Gym calls reuse the native endpoint's sampling check, prefix splice, token-field and routed-expert helpers, and the worker's capture lifecycle."
  • (C19) "vLLM bumps are not blocked by Dynamo … unit guards enforce both" holds for code only. Dynamo's dependency bounds (ai-dynamo-runtime caps pydantic at 2.13) can still hold a vLLM bump until Dynamo releases or an override is added; say so here too.

Performance (no action)

At the scale of the validation runs the step times are mostly noise (one engine, 1 to 10 steps, and native's own repeats vary about 6x), so they are not worth a table. One fact is: the R3 routed-experts payload adds no visible step time (median exposed generation / step time over 10 steps: 1.08 s / 4.29 s native, 1.09 s / 4.28 s Dynamo). Throughput at scale stays the follow-up the PR already lists.

Legacy entrypoints (FYI, no action)

Follow-ups (tracking, not for this PR)

  • Move the vLLM worker's token-capture lifecycle into its own object (a VllmRequestCapture in a new vllm/token_capture.py), as Megatron already has (megatron/token_capture.py#L14-L28). About 280 lines would leave vllm_worker_async.py, and the Dynamo engine's calls into private worker methods would drop from 10 to 3.
  • Session affinity covers only capture calls: Gym sends x-dynamo-session-id only from its capture handler (external_capture.py#L450-L455), although the ng_prefix splice on non-capture calls also happens after routing. Unmeasured; the fix is on the Gym side.
  • NeMo RL does not notice an engine that stops serving after registration: dynamo_endpoint_error is polled only during the registration wait. Dynamo logs the cause (etcd lease loss at ERROR, lease.rs#L45-L57), and the fleet just shrinks. A done-callback on the endpoint task that logs at ERROR would be cheap.
  • Upstream Dynamo issues to file:
    • Worker.run() calls configure_dynamo_logging() unconditionally (worker.py#L230), which removes the host process's root log handlers and sets the root level from DYN_LOG, and its vLLM log formatter ignores DYN_SDK_DISABLE_ANSI_LOGGING. Engines embedded in a host process need an opt-out.
    • A request with more than 4 stop strings gets a 500 instead of a 400, and the message says "less than 4" although 4 is accepted (openai.rs#L230-L233).

Generated by Claude Code

Comment thread docs/design-docs/dynamo-integration.md Resolved
Comment thread nemo_rl/distributed/virtual_cluster.py Resolved
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 08c50d2 (PR #4905 from jthomson04/dynamo-native-engine)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test 08c50d2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test 2ba2cae

@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 2ba2cae (PR #4905 from jthomson04/dynamo-native-engine)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Resolves the docker/Dockerfile conflict with NVIDIA-NeMo#4340 (uv cache mounts). The
BUILD_DYNAMO RUN keeps this branch's form: docker/dynamo/install.sh now only
downloads etcd and nats-server, so it is not a uv-writing RUN and takes no
uv cache mount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test b065eb6

@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: b065eb6 (PR #4905 from jthomson04/dynamo-native-engine)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Gym#4817 (jthomson04/dynamo-frontend-dialect) is rebased onto Gym main
and now sits on top of it; its head moves from a369e28 to a5f936c. The
only conflict was Gym main's new custom-tool-strategy validator, which
sits next to the Dynamo frontend validator; both are kept. Gym's
dependencies are unchanged, so uv.lock needs no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test 94c0e96

@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 94c0e96 (PR #4905 from jthomson04/dynamo-native-engine)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

@jthomson04
jthomson04 marked this pull request as ready for review October 26, 2026 22:04
@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Request changes for C23: enabling Dynamo makes a valid multi-shard Gym configuration fail during shard planning. C24 fixes the local-install instructions, and C25 is a low-severity maintainability suggestion.

The shared native request pipeline is a useful seam to retain. The current source contains the earlier fixes for request-specific token limits, vLLM client-error mapping, disconnected captures, per-engine IPC endpoints, and engine system ports; those findings are not repeated.

This was a broad static review of the changed paths and their affected integration paths. Tests, linters, image builds, and GPU runs were not executed. The Gym gitlink was assessed, but its upstream contents were not populated in the mounted tree. The configuration/build topic was independently re-inspected because its delegate's completed report was unavailable.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread nemo_rl/environments/nemo_gym.py Resolved
Comment thread docs/guides/dynamo-generation.md Resolved
…osition

From the NVIDIA-NeMo#4905 review:
- C23: setup_nemo_gym_config wrote the dialect as a top-level policy_model
  overlay, which a sharded Gym plan rejects as unowned. Setup now stashes it
  under a NeMo RL key (dynamo_dialect, alongside tokenizer_config), and
  _build_gym_actor_config applies it to each actor's composed config. A
  shard's own template kwargs still win over the native defaults.
- C24: the local install instructions export NEMO_RL_DYNAMO_BIN_DIR, so the
  run that follows finds etcd and nats-server.
- C25: the two in-function rationale blocks in native_engine.py are one line
  each; the design doc carries the full reasons.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test e182811

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 26, 2026 22:52
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: e182811 (PR #4905 from jthomson04/dynamo-native-engine)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Resolve examples/configs/grpo_math_1B.yaml: keep main's new
lora_refit_mode default and this branch's comment, which no longer lists
the removed dynamo backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test b498510

@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: b498510 (PR #4905 from jthomson04/dynamo-native-engine)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

C59. Dynamo's Worker.run() starts with configure_dynamo_logging(), which
replaces the whole actor's Python logging: it removes NeMo RL's root handler,
sets the root level from DYN_LOG (so NRL_LOG_LEVEL=DEBUG records are dropped),
and swaps vLLM's handlers for an always-ANSI formatter. start_dynamo_endpoint
now replaces that module global with a no-op before the Worker starts.

A test checks the replacement keeps the root handlers, and another fails if
Worker.run() stops calling the setup by that name, so a Dynamo bump cannot
silently undo it. ai-dynamo/dynamo#18447 asks for a WorkerConfig opt-out that
replaces this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test c6a4c66

terrykong
terrykong previously approved these changes Nov 9, 2026
docker/Dockerfile conflicted with NVIDIA-NeMo#4863, which ran docker/dynamo/install.sh
with a throwaway uv cache. On this branch install.sh only downloads etcd and
nats-server, so the RUN keeps calling it directly. test_prune_uv_cache.py's
Dockerfile test keeps its no-cache-mount check and drops the two lines that
asserted the throwaway cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test 4cbebb8

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 10, 2026 17:07
@terrykong
terrykong disabled auto-merge November 10, 2026 17:36
terrykong
terrykong previously approved these changes Nov 10, 2026
@terrykong
terrykong enabled auto-merge (squash) November 10, 2026 18:35
vllm_worker_async.py's import block and pyrefly.toml's project-includes
conflicted with NVIDIA-NeMo#4506 (prefix recovery): each side added lines at the same
spot. Both sides are kept.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test 9919396

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 11, 2026 01:18
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 11, 2026
…-gym

Brings in main's prefix recovery (NVIDIA-NeMo#4506) through NVIDIA-NeMo#4905.

This branch moved the worker's inline chat server into
vllm/chat_server.py, and NVIDIA-NeMo#4506 changed that inline server, so the
worker's conflicts keep this branch's calls into chat_server and
NVIDIA-NeMo#4506's server changes are applied there:
- the request's restore fields, and on_sampling_params from the
  request class the routes bind to the worker;
- set_max_tokens sets max_tokens when a request set neither field;
  the frontend's render lowers the budget only when the call set one;
- preprocess_chat resolves a restored call's saved prefix. The block
  is three functions (resolve_restored_call, classify_restored_terminal,
  serve_restored_terminal) so the Dynamo adapter can run the same steps;
- chat_completion_full_generator records each output, restores the
  reply's text and usage, and wraps the parser;
- the generation-cut route, and aborts that never block the loop
  behind a cut (abort_request_capture).

Other conflicts:
- pyrefly.toml, vllm/utils.py, the wiring test's imports: both sides.
- test_capture_route_tensors.py, test_vllm_token_capture_hosting.py:
  NVIDIA-NeMo#4506's side, which builds the worker with _init_token_capture_state.

What NVIDIA-NeMo#4506 changed under this branch's code:
- the Dynamo adapter and its test hosts use the capture registry
  that replaced _capture_calls;
- two of NVIDIA-NeMo#4506's tests build the route's request class, which is a
  pydantic model here: they construct it instead of calling it bare.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 11, 2026
…NVIDIA-NeMo#4905 + NVIDIA-NeMo#4506) into jthomson04/dynamo-1.7

Each frontend now takes NeMo-Gym's generation cuts on a port of its own.
FrontendHost.start picks the cut port on its node beside the HTTP port and
returns both URLs (FrontendUrls). ManagedDynamoRuntime.generation_cut_control_urls
lists one per frontend_urls entry, in the same order, and
VllmGeneration.generation_cut_control_urls passes them to Gym, which pairs
each with its base_url. publish_chat_capture sends the cut's bearer token and
timeout to every frontend's capture file with the rest of the capture config.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@terrykong
terrykong merged commit f488a0c into NVIDIA-NeMo:main Nov 11, 2026
176 of 178 checks passed
@terrykong terrykong mentioned this pull request Nov 11, 2026
3 of 4 tasks
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 12, 2026
main's squash of NVIDIA-NeMo#4905 has the same tree as 9919396, which this branch
already merged, so this records it as merged with no content change
(-s ours). A plain merge of main would report 21 false conflicts on the
lines this branch has edited since.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
lonexreb added a commit to lonexreb/RL that referenced this pull request Nov 12, 2026
NVIDIA-NeMo#4905 removed the separate dynamo generation backend, so the dynamo
branches this PR patched and their test are gone; dynamo now runs on
the vLLM path, which already merges hf_config_overrides.

Signed-off-by: Shubhankar Tripathy <reach2shubhankar@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 12, 2026
…NVIDIA-NeMo#4905 squash) into jthomson04/dynamo-1.7

Two conflicts, both resolved by keeping both sides:
- docs/guides/dynamo-generation.md: "Where the cut goes" keeps NVIDIA-NeMo#4979's
  deadline (each engine cuts up to that frontend's deadline,
  token_capture.control_timeout_s from the cut's arrival there) and this
  branch's one cut port per frontend, one per base_url.
- tests/unit/models/generation/test_dynamo_generation_cut.py: imports
  only; NVIDIA-NeMo#4979's late-engine test runs through this branch's per-host
  capture publishing.

With several frontends each frontend that takes a cut reads the timeout
from its own node's capture pickle, which publish_chat_capture writes on
every host, and so takes its own deadline. An engine runs the cuts from
different frontends one after another, each against its own deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
mxinO added a commit that referenced this pull request Nov 12, 2026
Resolve uv.lock by re-locking the branch lock against the merged
pyproject (adds ai-dynamo 1.5.1 from #4905; pydantic split matches main).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Meng Xin <164952785+mxinO@users.noreply.github.com>
mxinO added a commit that referenced this pull request Nov 12, 2026
Resolve uv.lock by re-locking the branch lock against the merged
pyproject (adds ai-dynamo 1.5.1 from #4905; pydantic split matches main).

Signed-off-by: Meng Xin <164952785+mxinO@users.noreply.github.com>
lauradang added a commit to lauradang/RL that referenced this pull request Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests CI Relating to CI Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants